Reducing the Network Load of Replicated Data

In the Internet today, transfer rates are often limited by the bandwidth of a bottleneck link rather than the computing power available at the ends of the links. To address this problem, we have designed a novel link layer protocol that employs computation at the ends of the link to reduce bandwidth consumption across the link. Our scheme is motivated by the prevalence of repeated transfers of the same information, as typically occurs in the form of HTTP, FTP, and DNS traffic. The protocol complements existing link compression and application-level caching schemes by combining aspects of both. It is able to detect and remove redundancy at the packet level and across large timescales. 'We make three contributions in this thesis. First, to motivate our scheme we show by packet trace analysis that there is significant replication of data at the packet level, mainly due to Web traffic. Second, we present an innovative link compression protocol that is well-suited to traffic with such long-range correlation. Third, we demonstrate by experimentation that the availability of inexpensive memory and general-purpose processors in PCs makes our protocol practical and useful at rates exceeding T3 (45 Mbps). Thesis Supervisor: John V. Guttag Title: Professor and Associate Head, Computer Science and Engineering Thesis Supervisor: David J. Wetherall Title: Research Assistant